Edge Based Cmy Automatic Picture Registration

ABSTRACT

An automatic process for performing CMY (Cyan, Magenta, Yellow) registration for film digital restoration. After three color components Cyan, Magenta, and Yellow of a picture are scanned into files, each component is divided into blocks, and edge detection is applied to each block, and an edge match is performed. The data of displacements is processed, and then an affine transform parameters are calculated. The affine transform is then applied for each block, and warping is used to combine the color components and obtain the registered picture of a color component.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional patentapplication Ser. No. 60/647,682, filed on Jan. 27, 2005, which isincorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to film preservation and restorations.More particularly, it relates to a method for automatically registeringthe color components (Cyan, Magenta, Yellow) of a color film for use inpreservation and restoration applications.

2. Description of the Prior Art

In order to store a color film for a long time and reduce the effect ofcolor fading, the color film is separated into three-color components,Cyan, Magenta, and Yellow (CMY). Each color component is stored on aseparate reel. When the time comes to re-release the film, the colorcomponents on each of these three reels need to be re-combined. In thisregard, the CMY components need to be registered to obtain resultingre-combined color images that appear to be the same color as the imageson the original color film. Most CMY registration is performed usingphoto-chemical techniques. Unfortunately, as the CMY reels age, the filmon each reel is subject to distortion or shrinkage. In this environment,such photo-chemical based CMY registration does not perform well. Assuch, it requires registration to be performed using digital technology.In this case, registration is performed manually. However, manualregistration is labor and cost intensive.

SUMMARY OF THE INVENTION

In accordance with the principles of the invention, a digital imageregistration technique automatically performs registration. In addition,the digital image registration technique can also register severelydistorted color components very accurately by warping images.

According to one embodiment, the method for automatically registeringthe color components of a color film includes determining correlationsbetween the color components of the color film, processing thecorrelation data, determining Affine Transform parameters for the colorcomponents, calculating the Affine Transform for each pixel in therespective color component using the determined parameters, andcombining the color components to re-produce the color film.

In order to determine the color component correlations, a base color isselected and initial displacement coefficient vector values of the othercolor components with respect to the selected base color component iscalculated. If necessary, each picture frame can be divided into blocks.The picture frames are edge detected for the respective colorcomponents, and the detected edges are matched for each color componentwith respect to the base color component. Once complete, newdisplacement vector values are calculated using the initially calculateddisplacement coefficient vector values.

According to a further embodiment, the method includes an errorcorrection aspect to the correlation processing stage. Initially, adetermination is made whether any large errors are present in thedetermined correlations. Any large errors are modified and newdisplacement value coefficients are calculated. The displacement vectorvalues are then re-calculated using the newly calculated displacementvalue coefficients. In order to calculate the new displacement valuecoefficients, a 3-order curve or 3-order plane is applied to best fitthe error numbers in either direction.

The modifying of errors can be performed using by using neighboringblock values or interpolation/extrapolation.

In order to calculate the Affine transform includes determining pixelposition in the original picture for each pixel in a block using anearest block analysis, and defining affine transform parameters usingthe determined pixel positions and displacement values of thecorresponding nearest blocks.

According to another embodiment, the combining is performed by warpingnon base color components with the base color component to form theregistered color image. The warping includes mapping each pixel in thenew picture onto the old picture using the calculated Affine Transform.

In yet another embodiment, the method for edge based CMY automaticpicture registration of a color film includes determining displacementvalues between a base color component and other color components of acolor film, and processing the correlation data to obtain newdisplacement value coefficients corresponding to the determineddisplacement values identify and remove errors. Once processing iscomplete, Affine Transform parameters are determined for the other colorcomponents, and the Affine Transform for each pixel in the respectivecolor component is calculated using the determined parameters. The colorcomponents are then combined to re-produce the color film.

Other aspects and features of the present invention will become apparentfrom the following detailed description considered in conjunction withthe accompanying drawings. It is to be understood, however, that thedrawings are designed solely for purposes of illustration and not as adefinition of the limits of the invention, for which reference should bemade to the appended claims. It should be further understood that thedrawings are not necessarily drawn to scale and that, unless otherwiseindicated, they are merely intended to conceptually illustrate thestructures and procedures described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings wherein like reference numerals denote similarcomponents throughout the views:

FIG. 1 is block diagram of the automatic color combination algorithmaccording to an embodiment of the invention;

FIG. 2 is an example of a block division according to one embodiment ofthe invention;

FIG. 3 is another example of a block division according to anotherembodiment of the invention;

FIG. 4 is an exemplary diagram of the edge matching method according toan embodiment of the invention;

FIGS. 5 a-5 f are a further example of the block division and relatedaffine transform displacement value application according to anembodiment of the invention;

FIG. 6 a-6 c are exemplary data sets used to represent the calculationof the affine transform according to an embodiment of the invention; and

FIG. 7 is a diagrammatic representation of the method for obtaining thewarping component picture using the calculated Affine Transformaccording to an embodiment of the invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The concepts of the present invention utilize known elements andprocesses in the film processing art. For example, the details of filmprocessing, affine transformations, CMY registration, etc., are wellknown and not described in detail herein. In addition, the invention maybe implemented using conventional programming techniques.

In a preferred embodiment, the invention is implemented in software.This invention can be, but is not limited to being, embedded infirmware, resident on microcomputer, microcode, etc. Other embodimentsmay be entirely hardware, entirely software, or a combination ofhardware and software elements.

Additionally, the present invention can be in the form of a softwareproduct stored or accessible from any computer usable medium providingcomputer program code. This includes, but is not limited to anyapparatus that may store, communicate, or propagate the program for useby or in connection with, any device capable of program execution. Themedium may be optical, electronic, magnetic, electromagnetic, atransmission medium, or a semiconductor medium. A computer readablemedia may be embodied as a computer hard drive, removable computer disk,random access memory, read only memory, semiconductor or solid statememory device, magnetic tape, punch card, or optical disk. Currentexamples of optical disks include Compact Discs (CDs), Digital VideoDiscs (DVDs), High Definition DVDs (HD-DVDs), Laserdiscs, Blu-Ray Discs,Minidiscs, or magneto-optical discs. With the exception of Laserdiscs,all of these disks may be in a fixed read only memory (ROM), recordable(±R), or recordable/rewriteable (-RW) format.

A data processing system may be comprised of one or more processors withsupporting electronic devices such as a motherboard. These processorsmay include memory resident on the processor or connected via a systembus to local memory, cache memory, or shared system or network memory.The data processing system may be coupled to input devices such askeyboards or mice, output devices such as displays or printers, andcommunications adapters such as network cards, modems, or networkingbackplanes.

Network adapters that may be included in a data processing system allowdata to be transferred across an intervening public or private networkto and from other terminals, servers, printers, or remote storagedevices. Some current examples of network adapters are Ethernetadapters, wireless WiFi and WiMax adapters, token ring adapters, etc.Current networks include Local Area Networks (LANs), Wide Area Networks(WANs), the Internet, ad hoc networks, direct connect networks orvirtual private networks (VPNs).

In accordance with the principles of the invention, the imageregistration process automatically performs registration for the Cyan,Magenta, and Yellow (CMY) color components in the digital domain. Thoseof ordinary skill in the art will recognize that concepts disclosedherein are not limited to C, M, and Y, and may be use for other colorspaces as well, or between any two color components.

A 3-reel CMY film is scanned into three mono sequences. The picture sizecan be 2K or 4K (one K is 1024 bytes). In the film industry, theresolution of 2K is 2048×1556 pixels and the resolution of 4K is4096×3112 pixels. The bit-depth of a pixel is irrelevant to the presentinvention and is typically 10 bits. The scanned files are illustrativelystored in a dpx format (SMPTE (Society of Motion Picture and TelevisionEngineers) Digital Picture Exchange Format). However, other file formatscan be used and supported without departing from the spirit of theinvention.

The registration process of the present invention operates on one frameof image data at a time. As described below, there may be circumstancesthat make it necessary to further divide the frame of image date intoblocks and, if possible, sub blocks or sub pictures in order to continuethe processing.

One factor that may require dividing the picture into blocks or subblocks can be distortion of the source image data. Depending on theseverity of distortion, the picture may require division into blocks(i.e., when the non-linearity of the distortion cannot be ignored). Ablock can have some overlap with its adjacent blocks or it can have nooverlap at all. The number of blocks is determined based on the contentsof the picture data, which number can be a very rough estimate beforeactually performing the block division.

Generally, increasing the accuracy of the registration process requiresmore blocks. However, increasing the number of blocks means each blockwill be smaller in size, and the smaller block size, the potential forlower accuracy in the calculated displacement accuracy is higher (i.e.,if the block is too small, there may not be enough information, thuslower accuracy).

According the principles of the present invention very small blocks arenot required to perform the automatic picture registration. Experimentalresults indicate that the number of blocks for 2K materials can be 1×1,2×2, 2×4, 4×4, 4×6, or 4×8, to just name a few. Although possible, nomore than 8 blocks in either dimension are should be required.

In order to register three (3) color components (e.g., CMY, RGB), thecorrelations between them need to be determined. There are several waysto calculate these correlations. Illustratively, edge correlation oredge matching is used. In this regard, there are two steps: edgedetection and edge matching. Any existing edge detection techniques canbe used, for example the Canny edge detection technique as known in theart. The edge matching is implemented after the edges are detected. Anyof the three colors components to be registered can be chosen as a base,and displacements (i.e., correlations) of the other two color componentscan be calculated from the chosen base component.

By way of example, there are two displacement vectors for each block,(V^(x) _(rg), V^(y) _(rg)), (V^(x) _(rb), V^(y) _(rb)), where V^(x)_(rg) is a displacement between red and green in the x direction, andV^(y) _(rg) is a displacement between red and green in the y direction.Similarly, V^(x) _(rb) and V^(y) _(rb) are displacements between red andblue in the x and y directions, respectively. Here we assume the redcomponent is used as the base.

In order to assist in the correlation determination, it is preferred todivide each frame into blocks. When the picture is divided into blocks,all the image data is better configured to be processed to eliminate bigerrors and to make the values of the displacements change smoothlyacross the picture. For example, a one-dimensional 3-order curve ineither the x or y direction can be used, or a two-dimensional 3-orderplane. Also, a lower order or a higher order curve or plane can be used.When the number of blocks in a direction is less than three, then nocurve fit is taken for the displacement values.

The adjusted values of displacement (parameters) are used to calculatesix parameters of an affine transform. Four displacement vectors areused to calculate the affine transform of each block, and redundancy canbe used to reduce errors. However, the present invention does notrequire the use of redundancy to reduce errors, but may use it to find apixel in a new picture to its corresponding location in the old picture,and obtain the pixel value by interpolating. The new picture is saved ina file format, such as dpx, yuv, raw, or ppm as known in the art.

An illustration of the automatic color combination process 10 is shownin FIG. 1 a. The inputs 12 a, 12 b, and 12 c are three separate colorcomponents. These color components can be stored in a single file orseparate files. In a preferred embodiment, the inputs are three separateDPX files, one for each color component.

There are potentially two options in the next step 14: 1) Edges aredetected first by an edge detector 16 and then the edge picture isdivided into blocks using a divider 18 (See FIG. 1 a); or 2) A picturecan be divided into blocks first using a divider 18 and then implementedge detection 16 (See FIG. 1 b). The resulting outputs for these twomethods however, may be different. Step 14 is the first step in thecorrelation determination of the present invention.

Dividers 18 utilize two parameters to divide the picture, 1) the numberof blocks in the horizontal direction, and 2) the number of blocks inthe vertical direction. As mentioned above, the blocks can beoverlapping or non-overlapping, where the portion of overlap can bevarying, and the block size can be different from one to another. If theblock size is varying, the size is determined by the contents of thepicture. The rich texture areas of the picture can have small blocks,and the less texture areas can have big blocks.

FIG. 2 shows an embodiment of four (4) overlapping blocks of fixed size.Note, area e is where two adjacent blocks overlap, and area f is whereall four blocks overlap. FIG. 3 shows an embodiment of four (4)non-overlapping blocks that vary in size.

As noted above, for performing edge detection, any existing edgedetector can be used, such as, e.g., the above-mentioned Canny edgedetector. All the edges are a single pixel thick for Canny edgedetection. Other edge detectors may have multi-pixel thick edges.

As the final part of the correlation determination process, thedivided/edge detected images are edge matched 20. For edge matching 20,a search window is opened on the base color edge picture. For eachposition in the search window, a non-base color edge block is comparedwith the base color. The number of unmatched edge points is calculated,and the smallest number is picked as the best match; or alternatively,the number of matched edge points is calculated and the largest numberis picked as the best match.

According to other embodiments, the best match may be tested to avoid amis-pick. One example of such a test is now described with reference toFIG. 4. The number of the mismatch edge points at the position a shouldbe less than the number of any of the eight (8) positions of b and d. Aloose test is where the mismatch number at the position a should be lessthan any number at the four (4) positions of d. Those of skill in theart will recognize that that the data set may be low pass filtered firstto obtain minimums or maximums, or simply to improve accuracy.

Once the edge matching is performed, additional data processing 22 ofthe image is required for the registration process. This data processingprovides an error correction/prevention stage, and further improvesaccuracy by using the newly calculated displacement values (vectors). Anillustrative technique for data processing 22 according with theprinciples of the invention is as follows. For each block, there are 2displacement vectors (i.e., x and y). Each displacement vectorrepresents a displacement between a color component edge map to the basecolor component edge map in the horizontal and vertical directions. Fora picture of m by n blocks with a fixed block size, there are four setsof data: V^(ij) _(1x), V^(ij) _(1y), V^(ij) _(2x), V^(ij) _(2y), whereij are the indices of a block, and m, n are the number of blocks in xand y directions, respectively.

Here V^(ij) _(1x) and V^(ij) _(1y) are used as examples to show how toprocess the data. It is illustratively assumed that m=5, n=5, and V^(ij)_(1x) is a 5×5 matrix.

Step 1: Use a pre-set threshold to find out if there are any big errornumbers.

Step 2: Modify the big error numbers by using their neighbor values orby interpolation/extrapolation.

Step 3: Use 3-order curves to best fit the numbers in either directionor use a 3-order plane. For a 3-order curve:

f(x)=a ₀ +a ₁ *x+a ₂ *x ² +a ₃ *x ³, and  (1)

f(y)=b ₀ +b ₁ *y+b ₂ *y ² +b ₃ *y ³  (2)

and for a 3-order plane:

f(x,y)=a ₀ +a ₁ *x+a ₂ *y+a ₃ *x ² +a ₄ *y ² +a ₅ *xy+a ₆ *x ³ +a ₇ *y ³+a ₈ *x ² *y+a ₉ *x*y ²  (3)

where a_(i), and b _(j) are coefficients of the polynomial curve orplane. Certainly, lower order or higher order can be used.

If 3-order curve is used, for each row of a matrix, [x₀, x₁, x₂, x₃,x₄], the corresponding polynomial coefficients are calculated asfollows:

X=[1x₀x₀ ²x₀ ³;1x₁x₁ ²x₁ ³;1x₂x₂ ²x₂ ³;1x₃x₃ ²x₃ ³;1x₄x₄ ²x₄ ³],  (4)

where X is a 5×4 matrix, the semicolon ‘;’ is a row separator and x_(i)is the position of the corresponding block i in x direction in the row.

F=[f(x ₀)f(x ₁)f(x ₂)f(x ₃)f(x ₄)],  (5)

where F is a vector and f(x₁) is the displacement of the correspondingblock i in the row.

A=[a₀a₁a₂a₃],  (6)

where A is coefficient vector and is initially unknown.

Then,

F=X*A,  and (7)

A=(X ^(T) X)⁻¹ X ^(T) F,  (8)

where X^(T)X is positive definition, and it is inversible.

Step 4: Re-calculate the displacement values of F by using thecoefficients A:

F′=X*A;  (9)

where F′ is a new data set that is used to replace the old one. Afterall rows are processed, a new matrix is created with all the new numbersof F's. The parameters at most outside positions may be furthermodified, such that their values are within a certain range of thevalues of the second most outside parameters of the matrix.

A 3-order plane can be calculated in a similar fashion, except thematrix is bigger. For example, matrix X is 25×10, F is 10×1, and A is10×1.

Once the data processing is complete, the Affine Transform for eachcolor component block needs to be calculated 24 (See FIG. 1). For eachcolor component, there are two data sets, one in the x-direction and theother in the y-direction. For each pixel position of a block, a nearestblock analysis is used to determine the pixel position in the originalpicture. The present invention is described using the four nearestneighbor blocks to make this determination. However, those of skill inthe art will recognize that the number of blocks used in a nearest blockanalysis is a matter of choice and can be higher or lower than the “fournearest neighbor” example described herein.

FIGS. 5 a-f show an example of this concept. A more general case isshown in FIG. 5 a where if a block (I) has eight neighbor blocks, subblock 11 will use the displacement values of blocks A, B, D, I todetermine the parameters of the affine transformation. It follows thatsub block 12 will use the displacement values of blocks B, C, E, and I,sub block 21 will use the displacement values of blocks D, F, I, G, andsub block 22 will use the displacement values of blocks E, H, G, and I.For other cases, if block I is located at a side or a corner of apicture (FIGS. 5 e & 5 f), then the respective sub block 11, 12, 21, 22will use its nearest three neighboring blocks and block I to calculatethe affine transform parameters.

FIGS. 6 a, 6 b and 6 c indicate the new pixel (x, y) in the middle of 4block centers (FIG. 6 a), at a picture corner (FIG. 6 b), and at oneside of the 4 block centers (FIG. 6 c), respectively.

The affine transform is shown below,

$\begin{matrix}{\begin{bmatrix}x_{old} \\y_{old}\end{bmatrix} = {{\begin{bmatrix}a_{0} & b_{0} & c_{0} \\a_{1} & b_{1} & c_{1}\end{bmatrix}\begin{bmatrix}x_{new} \\y_{new} \\1\end{bmatrix}}.}} & (10)\end{matrix}$

The positions of the 4 old points (block centers) are known (shown inFIG. 6), and the positions of the 4 points in the new picture can beobtained by adding the displacements to the corresponding points. Thereare 8 equations and six unknowns, as such, the 6 parameter affinetransform can be easily obtained. In some cases when more than twoequations are linear combinations of the other equations, the affinetransform is reduced to:

$\begin{matrix}{\begin{bmatrix}x_{old} \\y_{old}\end{bmatrix} = {{\begin{bmatrix}a_{0} & 0 & c_{0} \\0 & b_{1} & c_{1}\end{bmatrix}\begin{bmatrix}x_{new} \\y_{new} \\1\end{bmatrix}}.}} & (11)\end{matrix}$

For warping of a picture (step 28 FIG. 1), warping is applied to two ofthe three color components. More generally, if there are N components,warping will be applied to N-1 color components. Those of skill in theart will recognize that image warping is one kind of imagetransformations, which can be linear or non-linear transformation. Theone color component that is used for the base will not be warped (in theexemplary case of FIG. 1, M/G). The warping component picture isobtained by using the calculated affine transform to map each pixel inthe new picture onto the old picture (illustrated in FIG. 7) This isdone by combining (28) the two warping color components with the basecolor component to form the registered color image. The mapped pixel(m,n) in the old picture is normally not on the integer grid, however,the value of the pixel can be obtained by using interpolation or thenearest pixel value. Any of several known interpolation techniques, suchas bi-linear, bi-cubical, etc. may be used for this purpose. Afterwarping, the three color components can be converted to certain desiredfile or image formats, and form a registered color image.

The above-described automatic registration process has been tested on anumber of films with good results.

In order to speed up the registration process, the picture can be downsampled to a lower resolution, for example from 4K down sampled to 2K,and the best matched displacement vectors of each sub picture can becomputed at the lower resolution. Then the matched displacement vectorsare up-scaled to the original resolution, these vectors are used toperform the picture warping at original resolution, 4K.

In another case, in order to reduce the scanning cost, lower resolutionis used. Thus, the lower scanned resolution, the lower the cost. Themagenta channel can be scanned at the high resolution, for example 4K,and cyan and yellow channels can be scanned at lower resolution, forexample 2K. The magenta channel is the most dominant of the threechannels, thus enabling this different resolution approach to the lessdominant cyan and yellow channels. This process effectively up-scalesthe cyan and yellow channels to the resolution of the magenta channel.Then the registration can be done in high resolution.

It should also be noted that the above-described registration processcan occur at the original separation of a color film into the CMYcomponents as a check on the quality of the separation, e.g., to checkif a component of a picture is missing or is damaged, etc.

In view of the above, the foregoing merely illustrates the principles ofthe invention and it will thus be appreciated that those skilled in theart will be able to devise numerous alternative arrangements which,although not explicitly described herein, embody the principles of theinvention and are within its spirit and scope. It is therefore to beunderstood that numerous modifications may be made to the illustrativeembodiments and that other arrangements may be devised without departingfrom the spirit and scope of the present invention as defined by theappended claims.

1. A method for automatically registering the color components of acolor film, the method comprising the steps of: determining correlationsbetween the color components of the color film; processing thecorrelation data; determining Affine Transform parameters for the colorcomponents; calculating the Affine Transform for each pixel in therespective color component using the determined parameters; andcombining the color components to re-produce the color film.
 2. Themethod according to claim 1, wherein said determining correlationscomprises: selecting a base color component; calculating initialdisplacement coefficient vector values of the other color componentswith respect to said selected base color component; dividing eachpicture frame into blocks; detecting picture frame edges for the colorcomponents; and matching the detected edges of each color component withrespect to a chosen base color component.
 3. The method according toclaim 2, wherein said processing of the determined correlationscomprises calculating new displacement vector values using the initiallycalculated displacement coefficient vector values.
 4. The methodaccording to claim 2, wherein said processing of the determinedcorrelations comprises: determining whether any large errors are presentin the determined correlations; modifying any large errors; calculatingnew displacement value coefficients; and re-calculating the displacementvector values using the newly calculated displacement valuecoefficients.
 5. The method according to claim 4, wherein saiddetermining is performed using a predetermined threshold.
 6. The methodaccording to claim 4, wherein said modifying errors is performed usingby using neighboring block values or interpolation/extrapolation.
 7. Themethod according to claim 4, wherein said calculating new displacementvalue coefficients is performed by applying a 3-order curve or 3-orderplane to best fit numbers the error numbers in either direction.
 8. Themethod according to claim 2, wherein said calculating the Affinetransform comprises: determining pixel position in the original picturefor each pixel in a block using a nearest block analysis; and definingaffine transform parameters using the determined pixel positions anddisplacement values of the corresponding nearest blocks. 9-20.(canceled)
 21. The method according to claim 1, wherein said combiningcomprises warping non base color components with the base colorcomponent to form a registered color image.
 22. The method according toclaim 21, further comprising converting the color components to adesired file or image format before forming the registered color image.23. The method according to claim 1, further comprising: downsamplingimage data to be registered to a lower resolution to speed upprocessing; and upscaling matched displacement vectors to a desiredresolution before said combining is performed.
 24. A method for edgebased CMY automatic picture registration of a color film, the methodcomprising the steps of: determining displacement values between a basecolor component and other color components of a color film; processingthe correlation data to obtain new displacement value coefficientscorresponding to the determined displacement values identify and removeerrors; determining Affine Transform parameters for the other colorcomponents; calculating the Affine Transform for each pixel in therespective color component using the determined parameters; andcombining the color components to re-produce the color film.
 25. Themethod according to claim 24, wherein said processing further comprisesre-calculating displacement vector values using the newly calculateddisplacement value coefficients.
 26. The method according to claim 24,wherein said determining correlations comprises: calculating initialdisplacement coefficient vector values of the other color componentswith respect to said selected base color component; dividing eachpicture frame into blocks; detecting picture frame edges for the colorcomponents; and matching the detected edges of each color component withrespect to a chosen base color component.
 27. The method according toclaim 26, wherein said determining Affine Transform parameterscomprises: determining pixel position in the original picture for eachpixel in a block using a nearest block analysis; and defining the affinetransform parameters using the determined pixel positions anddisplacement values of the corresponding nearest blocks.
 28. The methodaccording to claim 24, wherein said combining comprises warping theother color components with the base color component to form aregistered color image.
 29. The method according to claim 28, whereinsaid warping comprises mapping each pixel in the new picture onto theold picture using the calculated Affine Transform.
 30. The methodaccording to claim 27, further comprising converting the colorcomponents to a desired file or image format before forming theregistered color image.
 31. The method according to claim 24, furthercomprising: downsampling image data of the color film-to be registeredto a lower resolution; and upscaling matched displacement vectors to adesired resolution before said combining is performed.